"The highlighted tokens are primarily morphemes, syllables, or short word fragments in various languages, often marking grammatical functions, word formation, or key semantic units. These include suffixes, prefixes, and root components that are essential for constructing meaning, indicating tense, plurality, comparison, or other grammatical relationships. The activations focus on these subword units as they are crucial for understanding and generating morphologically rich or agglutinative languages, as well as for tokenization in multilingual contexts."
Score Type | Accuracy | Precision | Recall | F1 score | TPR | TNR | FPR | FNR |
---|---|---|---|---|---|---|---|---|
detection | 0.5 | 0.5 | 1.0 | 0.667 | 1.0 | 0.0 | 1.0 | 0.0 |
fuzz | 0.52 | 0.51 | 1.0 | 0.676 | 1.0 | 0.04 | 0.96 | 0.0 |